Correcting Image Capture Distortion

ABSTRACT

The present disclosure relates generally to improving readability of machine-readable information, despite signal capture distortion. One claim recites an apparatus comprising: memory for storing input data; an electronic processor programmed for: altering a digital watermark or a digital watermarking process to pre-distort a digital watermark signal, wherein the altering is intended to counteract or compensate for expected distortion so as to enable machine-based detection of an embedded, pre-distorted digital watermark signal despite the expected distortion, wherein the expected distortion includes distortion associated with optical capture of the input data; and embedding the pre-distorted digital watermark signal in the input data. Of course, other claims and combinations are provided as well.

RELATED APPLICATION DATA

This application is a division of U.S. patent application Ser. No.12/109,573, filed Apr. 25, 2008 (published as U.S. 2008-0298632 A1),which claims the benefit of U.S. Patent Application No. 60/913,987,filed Apr. 25, 2007. Each of the above patent documents is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to data hiding and digitalwatermarking. In some implementations the present disclosure relates tocorrecting optical scan or other capture distortion, e.g., such as lensblurring, focus errors and other scanning anomalies.

BACKGROUND AND SUMMARY

Digital watermarking—a form of steganography—is a process for modifyingmedia content to embed a machine-readable code into the content. Thecontent may be modified such that the embedded code is imperceptible ornearly imperceptible to the user, yet may be detected through anautomated detection process. Most commonly, digital watermarking isapplied to media such as images, audio signals, and video signals.However, it may also be applied to other types of data, including textdocuments (e.g., through line, word or character shifting, backgroundtexturing, etc.), software, multi-dimensional graphics models, andsurface textures of objects.

Digital watermarking systems typically have two primary components: anembedding component (or encoder) that embeds a watermark in mediacontent, and a reading component (or detector) that detects and readsthe embedded watermark. The embedding component embeds a watermark byaltering data samples of the media content in the spatial, temporal orsome other domain (e.g., Fourier, Discrete Cosine or Wavelet transformdomains). The reading component analyzes target content to detectwhether a watermark is present. In applications where the watermarkencodes information (e.g., a plural-bit message), the reader extractsthis information from the detected watermark.

The present assignee's work in steganography, data hiding and digitalwatermarking is reflected, e.g., in U.S. Pat. Nos. 5,862,260, 6,408,082,6,614,914 and 7,027,614, which are each hereby incorporated byreference. A great many other approaches are familiar to those skilledin the art. The artisan is presumed to be familiar with the full rangeof literature concerning steganography, data hiding and digitalwatermarking.

In some cases digital watermarking and other machine-readable indicia(e.g., barcodes, data glyphs, etc.) may be detected from optical scandata, examples of which are disclosed in, e.g., U.S. Pat. Nos.5,978,773, 6,522,770, 6,681,028, 6,947,571 and 7,174,031, which are eachhereby incorporated by reference.

Today's camera phones and other handheld readers present expandeddecoding opportunities—and challenges.

One challenge is providing handheld cameras (e.g., in a cell phone orother mobile device) to an army of users, with nearly each user having adifferent idea on proper focal length for image and video capture.

In some cases a user may want to be close to a marked image or video to,e.g., capture high spatial frequency content (see FIG. 1); but the closepositioning often results in a captured image that is slightly out offocus (or blurred)—which may hamper detection of a machine-readable codecontained or represented in the image.

Thus, one inventive combination provides a method including: obtaininginput data; altering a digital watermark or a digital watermarkingprocess to pre-distort a digital watermark signal, wherein the alteringis intended to counteract or compensate for expected distortion enablingmachine-based detection of an embedded, pre-distorted digital watermarksignal despite the expected distortion; and embedding the pre-distorteddigital watermark signal in the input data.

Another inventive combination provides a method including: obtaining aplurality of different models representing different expected distortionassociated with a plurality of different data captures, the differentdata captures each resulting in distortion of a machine-readable signal;indexing the different models; upon receiving a request, selecting amodel associated with the request; and providing the selected model.

Still another inventive combination provides a method including:obtaining input data; altering a digital watermark or a digitalwatermarking process to pre-distort a digital watermark signal, whereinthe altering is intended to counteract or compensate for expecteddistortion due to image capture or an image capture device, the alteringenabling machine-based detection of an embedded, pre-distorted digitalwatermark signal despite the expected distortion; and embedding thepre-distorted digital watermark signal in the input data.

Yet another inventive combination provides a method including: obtaininginput data, the input data representing imagery captured with at leastan optical lens, the input data comprising test data and amachine-readable signal; evaluating characteristics associated with thetest data to determine information regarding lens blurring of the inputdata associated with the optical lens; adjusting the input data tocompensate for or to correct the lens blurring based at least in part onthe information; and analyzing the compensated for or corrected inputdata to obtain the machine-readable signal.

Another inventive combination includes: obtaining input data, the inputdata representing imagery or video, the input data comprising test dataand a machine-readable signal; determining characteristics associatedwith the test data to determine information regarding signal capturedistortion of the input data; based on at least the characteristics,determining an amount of correction or counteracting to be applied tothe input data; applying a determined amount of correction orcounteracting to the input data; and analyzing corrected or counteractedinput data to obtain the machine-readable signal.

Further combinations, aspects, implementations, features, embodimentsand advantages will become even more apparent with reference to thefollowing detailed description, the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates image capture with a cell phone including an opticalsensor.

FIG. 2A illustrates an example target image.

FIG. 2B illustrates examples of adverse effects on focus (e.g.,blurring) of the FIG. 2A example target image.

FIG. 3 illustrates a plot diagram.

FIG. 4 is a block diagram illustrating an embedding process.

FIG. 5 is a diagram showing an approximation or estimate of lensblurring.

FIG. 6 is a correction template or filter to compensate for the blurringshown in FIG. 5.

FIG. 7 is a blurred signal (FIG. 5) after compensation by the FIG. 6correction template or filter.

FIG. 8A is a diagram showing test data in a spatial frequency domain(with acceptable lens blur); FIG. 8B is another diagram showing testdata in a spatial frequency domain (with unacceptable lens blur).

FIG. 9 shows a digital watermarked gray patch.

FIG. 10A shows the watermarked gray patch of FIG. 9 after capturedistortion (e.g., lens blurring).

FIG. 10B shows the FIG. 10A patch after compensation or correction(e.g., using the FIG. 6 correction).

DETAILED DESCRIPTION

When an image is distorted (e.g., blurred, out of focus, etc.) by anoptical capture device (e.g., a cell phone camera, PDA, digital camera,etc.) the loss of information from a target image (FIG. 2A) can be inthe form shown in FIG. 2B. For example, in some cases there may be aloss of information in some high spatial frequencies.

The distortion may result in some high spatial frequencies of the targetimage being “out of phase” or otherwise distorted.

In the case of a target image including or representing machine-readableinformation, e.g., a barcode, data glyph or digital watermark, somehigher spatial frequency information corresponding to at least some ofthe machine-readable information is also out of phase, making reliabledetection of the machine-readable information more difficult.

Distortion such as blurring can be modeled or approximated by aconvolution of a target image with, e.g., a Bessel function, aspostulated by J. I. Yellott et al., “Correcting spurious resolution indefocused images,” SPIE 6492, pp. 649200-1-649200-12 (2007), herebyincorporated by reference. FIG. 3 illustrates a portion of a Besselfunction, where the negative areas (shown with dotted boxes) correspondto a 90 degree phase shift.

Some examples are now provided illustrating and describing compensationfor distortion.

In a first implementation a target image includes a digital watermarksignal. But prior to being embedded in the target image, the digitalwatermark signal is pre-distorted with a phase shift, e.g., shifted 90degrees relative to expected distortion. For example, if the distortionis likely to occur in high frequency areas such as discussed above withrespect to convolving with the function shown in FIG. 3, high frequencycomponents of the digital watermark are shifted 90 degrees. (Anappropriate shift can be achieved by, e.g., convolving a digitalwatermark signal with an appropriate or corresponding Bessel function.)The pre-distorted watermark is then embedded into an image and printed,engraved or otherwise reproduced. Then, when expected distortion occurs(e.g., blurring due to image capture or focus errors), the highfrequency watermark components are distorted (e.g., convolved) into amore readable form, enabling better watermark detection.

As another example, if a digital watermark includes signal elements withvalues corresponding to [+1, 0, −1] at positions in a high frequencyarea, the signal is preferably phase shifted to compensate for excepteddistortion. In the case of distortion modeled by a convolution with atarget image and the function shown in FIG. 3, the signal is shifted 90degrees resulting in signal elements with values corresponding to [−1,0, +1] at the positions in the high frequency area. When thispre-distorted digital watermark signal is subjected to blurring duringimage capture, the pre-distorted signal elements are distorted again—butin an expected manner—resulting in signal elements with valuescorresponding to [+1, +1, 0 −1], which correspond to the original signalelements. (The blurring shifts the signal essentially back to itsoriginal, pre-distorted form.)

FIG. 4 illustrates a block diagram corresponding to one implementationof pre-distortion.

An image 10 is obtained to be watermarked. The image 10 can be of anyform, e.g., a color image, grayscale, a photograph, a graphic orartwork, video representation, etc. If the image 10 is in analog form,e.g., a printed image, it can be optically scanned or captured (e.g.,with a digital camera or optical sensor) or otherwise converted into adigital image.

A message 12 is provided (e.g., 1 or more bits). The message 12 isintended to be hidden within the image 10 with digital watermarking orsteganography. A digital watermarking signal is generated by a digitalwatermark encoder 20 to represent or otherwise carry the message 12. Thedigital watermark signal may be an additive signal, e.g., one that isadded (or subtracted, multiplied, etc.) to the image 10, may representinstructions or changes that should be made to the image 10 to carry themessage, e.g., based on a key, and/or may include changes ormodifications to frequency domain coefficients, and/or may include arandom or pseudo-random component. Of course, other digital watermarkingtechniques may be used as well.

The digital watermark signal is pre-distorted to compensate for expecteddistortion, e.g., by a distortion module 22. Distortion module 22 may beincorporated into encoder 20 or encoder 20 may other wise communicate orcooperate with a distortion module 22. Distortion module 22 accesses orotherwise determines a distortion model 24. Distortion model 24 providesa template, mask or other instructions to be used by distortion module22 when pre-distorting the digital watermark signal. The distortionmodel 24 can be based on or tailored to, e.g., the type of image 10, thewatermark message 12, the expected distribution channel through whichimage 10 will travel, the expected type of image capture, optical lenssystem or a model of distortion introduced by such, digital watermarkencoder 20, human visual system (HVS), etc. In one implementation, thedistortion module 22 (or watermark encoder 20) communicates with adatabase, model library or a network remote resource to access anappropriate model 24. The term “appropriate” in this context impliesthat a model 24 is selected or obtained to compensate for expecteddistortion. Of course, the distortion module 22 may be pre-programmedwith a default distortion model 24, e.g., based on or tailored toexpected distortion that a digital watermark will most likely encounter.

Prior to embedding the digital watermark signal in the image 10, thedistortion module 22 accesses a distortion model 24 and pre-distorts adigital watermark signal to counteract or compensate for expecteddistortion associated with optical scanning or capture. Returning to anexample discussed above, e.g., blurring or other distortion that can bemodeled with convolving image 10 with a Bessel Function, the digitalwatermark is phase shifted in some or all higher frequencies, e.g., 90degrees, to compensate for the expected distortion.

The pre-distorted digital watermark—representing or carrying message12—is embedded in image 10 to yield a watermarked image 14. Thewatermarked image 14 can then be printed or otherwise reproduced in ananalog form.

A printed (or otherwise fixed) image 16 that includes a pre-distorteddigital watermark signal can then be optically scanned or captured—whichintroduces expected distortion. But since the digital watermark signalhas been pre-distorted, the expected distortion transforms (e.g.,modeled as a convolution with a determined function) the pre-distorteddigital watermark signal into a more readable form.

Optical data corresponding to the printed (or otherwise fixed),watermarked image 16 is provided to a digital watermark decoder toanalyze the scan data and recover the message 12. Of course, the digitalwatermark decoder may be co-located with an image sensor or otherwiselocated on a handheld device (e.g., a cell phone or personal digitalassistant (PDA)). Still further, the digital watermark decoder may beremotely located from an image capture device.

Post-Capture Correction

In other embodiments, a digital watermark is provided without thepre-distortion discussed above. Correction for lens blur is appliedafter image capture. These post-capture correction embodiments may allowfor an even more imperceptible watermark, relative to a pre-distortionwatermark.

While post-capture creation may include some post-processing, it may bemore flexible relative to the pre-distortion methods and systemsdiscussed above, as many types or amounts of capture distortion can becorrected.

A digital watermark is embedded in a host signal (e.g., an image orvideo). The embedded host signal is provided onto a surface (projectedor rendered if video) or object (e.g., printed, engraved, etc.) and thenan optical scan or digital image is captured of the surface or object.

Recall, here, the digital watermark has not been pre-distorted asdiscussed above with respect to FIG. 4. So prior to watermark detection(or as an initial stage of the watermark detection) the captured imagedata is filtered or corrected. The filter adjusts the captured imagedata to compensate for the capture distortion (e.g., lens blur).

To determine an amount or level of compensation or correction, oneprocess quantifies or otherwise evaluates image blur in the image orvideo data. Such an evaluation or quantification helps to determine anappropriate amount of correction to be applied to captured image andvideo.

In one example, test data can be introduced into an image or video.Characteristics associated with the test data can be evaluated incaptured imagery to help determine an amount or level of lens blur.

One example of test data is white noise (or other type of noise, e.g., apseudo- random pattern, noise in some predetermined frequencies, etc.),which can be added to an image or video, e.g., as a part of orbefore/after digital watermark embedding. The white noise is preferablyimperceptible in the image or video, but of course, there may be someapplications where some perceptibility is allowed. After image or videocapture, the white noise can be evaluated, e.g., in a spatial frequencydomain as shown in FIGS. 8A and 8B.

FIG. 8A illustrates a spatial frequency response for captured imagerywith acceptable capture distortion. The term “acceptable” can bequantified by determining white noise signal magnitude at apredetermined spatial frequency or range of frequencies. If the signalmagnitude is at or above a predetermined magnitude for the predeterminedspatial frequency (or range of frequencies) the capture distortion canbe deemed acceptable.

FIG. 8B illustrates a spatial frequency response for captured imagerywith unacceptable capture distortion. The term “unacceptable” can bequantified by determining white noise signal magnitude at apredetermined spatial frequency or range of frequencies. If the signalmagnitude is below a predetermined magnitude for the predeterminedspatial frequency (or range of frequencies) the capture distortion canbe deemed unacceptable.

The signal magnitude level can also be used to help determine orestimate an amount or level of correction or adjustment that is neededas well. For example, if the magnitude is below (or between) a firstpre-determined amount, a first level is determined; and if the magnitudeis below (or between) a second pre-determined amount, a second level isdetermined An amount or level of correction or adjustment correspondingto the first pre-determined amount or the second pre-determined amountis then selected.

Another example of test data are predetermined structures. For example,a dark spot surrounded by a lighter (e.g., white) color is provided inthe spatial domain in an image or video. The dark spot and surroundinglighter color are preferably subtle so as not to detract from the imageor video. A spatial frequency plot of captured imagery including suchtest data is useful in evaluating capture distortion such as lens blur.The dot structure is approximated by a delta function in a spatialfrequency domain. The magnitude and frequency location of the deltafunction can be used to determine an amount, type or level of capturedistortion. Appropriate correction can be determined and applied tocaptured imagery if needed.

Once an estimate of the lens blur is determined, it can be used to helpcorrect capture distortion in captured imagery. For example, theestimate may be mapped against pre-determined levels of lens blurring,e.g., stored in a data structure associated with or cooperating with adigital watermark detector. In one implementation, estimated distortionis used to determine a level, type or degree of lens blurring or otherdistortion. A corresponding scale factor or other correspondingmodification can then be obtained, e.g., from a data structure. Anobtained scale factor or other modification can then be applied to amaster correction template or filter.

An approximation of one example lens blur is shown in FIG. 5, wherepixel magnitude of the blur is illustrated. The illustrated blur spansmore than 15 pixels, which would severely disrupt an image or video inthe pixel area. A master correction template or filter is shown in FIG.6, e.g., for a 1-dimensional signal. (Of course, other types of signals,e.g., 2-dimensional signal can be similarly corrected.) The mastercorrection template or filter is adjusted, scaled or modified by adetermined scale factor or other corresponding modification, if needed.The adjusted, scaled or modified template or filter is then applied tothe captured imagery to counteract, adjust or correct image blur (FIG.7). A digital watermark detector (or other machine-readable symbologydetector) analyzes the counteracted, adjusted or corrected capturedimagery to read or detect a digital watermark (or other machine-readablecode).

In another implementation, a correction template or filter is generatedupon evaluation of image blur. For example, based on an evaluation oftest data in a spatial frequency domain, a correction filter isgenerated to counteract lens blur. A generated correction template orfilter is applied to captured imagery to counteract, adjust or correctimage blur. A digital watermark detector (or other machine-readablesymbology detector) analyzes the counteracted, adjusted or correctedcaptured imagery to read or detect a digital watermark (or othermachine-readable code).

To further illustrate some of these processes, an original watermark wascreated by embedding a Digimarc Mobile digital watermark signal(provided by Digimarc Corporation headquartered in Beaverton, Oregon)into a gray image patch. The digital watermark signal is readilydetectable from the watermarked gray image patch (FIG. 9). Lens blur(e.g., approximated by FIG. 5) is applied to the watermarked gray patch(FIG. 9), resulting in a blurred image patch shown in FIG. 10A. Thedigital watermark is difficult (and perhaps impossible) to read from thedistorted FIG. 10A patch. But, after compensation (e.g., as approximatedin FIG. 6) is applied to the distorted FIG. 10A image, e.g., as shown inFIG. 6, the watermark is readily detected from the resulting image shownin FIG. 10B. My tests show that the compensated for gray patch (FIG.10B) reads about 5 dB better than the FIG. 10A patch and approaches thedetection results of a digital watermark embedded in the original grayimage patch (FIG. 9).

Concluding Remarks

Having described and illustrated the principles of the technology withreference to specific implementations, it will be recognized that thetechnology can be implemented in many other, different, forms.

The methods, processes, components, modules, generators and systemsdescribed above may be implemented in hardware, software or acombination of hardware and software. For example, digital watermarkembedding or decoding processes may be implemented in a programmablecomputer or a special purpose digital circuit. Similarly, thesewatermarking processes may be implemented in software, firmware,hardware, or combinations of software, firmware and hardware.

The methods, components and processes described above may be implementedin software programs (e.g., C, C++, Visual Basic, Java, executablebinary files, etc.) executed from a system's memory (e.g., a computerreadable medium, such as an electronic circuitry and/or an optical ormagnetic storage device).

The section headings are provided for the reader's convenience. Featuresfound under one heading can be combined with features found underanother heading. Of course, many other combinations are possible giventhe above detailed and enabling disclosure.

We have used the terms “image” and “imagery” above. Both of these termsshould be construed broadly enough to include images or video.

The particular combinations of elements and features in theabove-detailed embodiments are exemplary only; the interchanging andsubstitution of these teachings with other teachings in this and theabove-mentioned U.S. patent documents are also contemplated.

1. A method comprising: obtaining input data; using a programmedprocessor, altering a digital watermark or a digital watermarkingprocess to pre-distort a digital watermark signal, wherein the alteringis intended to counteract or compensate for expected distortion so as toenable machine-based detection of an embedded, pre-distorted digitalwatermark signal despite the expected distortion, wherein the expecteddistortion includes distortion associated with optical capture of theinput data; and embedding the pre-distorted digital watermark signal inthe input data.
 2. The method of claim 1 further comprising printing orengraving the input data including the pre-distorted digital watermarksignal.
 3. The method of claim 1 wherein the input data comprises datarepresenting an image or video.
 4. The method of claim 1 wherein thealtering utilizes a model of the distortion resulting from opticalcapture.
 5. The method of claim 1 wherein the expected distortionincludes blurring resulting from optical capture of the printed orengraved input data.
 6. The method of claim 5 wherein the blurring isassociated with lens focal length.
 7. The method of claim 5 wherein theblurring—left uncompensated for—would result in a loss of high frequencyinformation of the digital watermark signal upon signal detection. 8.The method of claim 1 wherein said act of altering alters a phase of thedigital watermark signal relative to the expected distortion.
 9. Themethod of claim 8 wherein an altered phase of the digital watermarksignal causes the digital watermark signal to be less readable in theabsence of the expected distortion.
 10. The method of claim 1 whereinexpected distortion is modeled as a convolution of the input data with apredetermined function.
 11. The method of claim 10 wherein thepredetermined function is approximated by a Bessel function.
 12. Themethod of claim 1 wherein the expected distortion is associated withlens focal length.
 13. A non-transitory computer readable mediumcomprising instructions or circuitry to perform the method of claim 1.14. An apparatus comprising: memory for storing input data; anelectronic processor programmed for: altering a digital watermark or adigital watermarking process to pre-distort a digital watermark signal,wherein the altering is intended to counteract or compensate forexpected distortion so as to enable machine-based detection of anembedded, pre-distorted digital watermark signal despite the expecteddistortion, wherein the expected distortion includes distortionassociated with optical capture of the input data; and embedding thepre-distorted digital watermark signal in the input data.
 15. Theapparatus of claim 14 in which the electronic processor controlsprinting or engraving of the input data including the pre-distorteddigital watermark signal.
 16. The apparatus of claim 14 wherein theinput data comprises data representing an image or video.
 17. Theapparatus of claim 14 wherein the altering utilizes a model of thedistortion resulting from optical capture.
 18. The apparatus of claim 14wherein the expected distortion includes blurring resulting from opticalcapture of the printed or engraved input data.
 19. The apparatus ofclaim 18 wherein the blurring is associated with lens focal length. 20.The apparatus of claim 18 wherein the blurring—left uncompensatedfor—would result in a loss of high frequency information of the digitalwatermark signal upon signal detection.
 21. The apparatus of claim 14wherein the altering alters a phase of the digital watermark signalrelative to the expected distortion.
 22. The apparatus of claim 21wherein an altered phase of the digital watermark signal causes thedigital watermark signal to be less readable in the absence of theexpected distortion.
 23. The apparatus of claim 14 wherein expecteddistortion is modeled as a convolution of the input data with apredetermined function.
 24. The apparatus of claim 23 wherein thepredetermined function is approximated by a Bessel function.
 25. Theapparatus of claim 14 wherein the expected distortion is associated withlens focal length.